Skip to content

Fix undefined signed shifts in odd-bit SIMD kernels - #67

Merged
gouyt13 merged 1 commit into
mainfrom
fix/odd-bit-simd-shift-ub
Aug 23, 2026
Merged

Fix undefined signed shifts in odd-bit SIMD kernels#67
gouyt13 merged 1 commit into
mainfrom
fix/odd-bit-simd-shift-ub

Conversation

@gouyt13

@gouyt13 gouyt13 commented Aug 23, 2026

Copy link
Copy Markdown
Member

Summary

  • load packed high-bit planes as uint64_t without alignment or aliasing assumptions
  • perform the 3-, 5-, and 7-bit unpacking shifts with unsigned semantics in both AVX2 and AVX-512 kernels
  • add a backend-independent regression test that exercises bit 63 and compares both SIMD implementations with a scalar inner product
  • remove unused includes and replace two test VLAs with standard fixed-size arrays

Cause

The packed 64-bit high-bit plane was loaded as int64_t. When bit 63 was set, the value became negative and expressions such as top_bit << 1 invoked undefined behavior under C++17, which UBSan reported at runtime.

Verification

  • Release C++ tests: 22/22 passed
  • ASan + UBSan tests: 22/22 passed
  • Python tests: 54/54 passed
  • clang-format, clang-tidy, Ruff, ShellCheck, and git diff --check: passed

GIST performance

Pinned single-thread IVF searches over all 1,000 GIST queries, k=10, nprobe=50, seven repetitions:

Total bits Extra bits Main QPS Fixed QPS Change
4 3 1107.15 1116.90 +0.88%
6 5 1105.07 1095.25 -0.89%
8 7 1097.26 1092.31 -0.45%

All result IDs and distances were exactly identical, with maximum distance delta 0.0. The sub-1% QPS variation is consistent with measurement noise. The AVX2 and AVX-512 object text sizes are unchanged, and the helper copies are fully inlined.

@gouyt13 gouyt13 self-assigned this Aug 23, 2026
@gouyt13
gouyt13 merged commit 088f2de into main Aug 23, 2026
12 checks passed
@gouyt13
gouyt13 deleted the fix/odd-bit-simd-shift-ub branch August 24, 2026 16:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant